The present invention relates to frame relay networks protocols and, in particular, to systems which incorporate the functionalities described in the Frame Relay Forum Multi-Link Frame Relay UNI/NNI Implementation Agreement (FRF.16.1), the entire disclosure of which is incorporated herein by reference for all purposes. More specifically, the present invention provides techniques by which physically relocated serial links may be automatically resynchronized in the multi-link context.
Most networks are organized as a series of hardware and software levels or “layers” within each node. These layers interact to format data for transfer between, e.g., a source node and a destination node communicating over the network. Specifically, predetermined services are performed on the data as it passes through each layer and the layers communicate with each other by means of the predefined protocols. This layered design permits each layer to offer selected services to other layers using a standardized interface that shields those layers from the details of actual implementation of the services.
FIG. 1 illustrates the relationship of subnetworks and gateways to layered protocols. Assume that the user application 102A in host A sends an application protocol data unit (PDU) to an application layer protocol 102B in host B such as, for example, a file transfer system. The file transfer software performs a variety of functions and sends file records to the user. In many systems, the operations at host B are known as server operations and the operations at host A are know as client operations.
As indicated by the downward arrows in the protocol stack at host A, this unit is passed to the transport layer protocol 104A, which performs a variety of operations and adds a header to the PDU passed to it. At this point, the unit of data is often referred to as a segment. The PDU from the upper layers is considered to be data to the transport layer.
Next, the transport layer passes the segment to the network layer 106A, also called the IP layer, which again performs specific services and appends a header. This unit (now called a datagram in internet terms) is passed down to the lower layers. Here, the data link layer adds its header as well as a trailer, and the data unit (now called a frame) is launched into subnetwork 110 by the physical layer 108A. Of course, if host B sends data to host A, the process is reversed and the direction of the arrows is changed.
Internet protocols are typically unaware of what goes on inside the network. The network manager is free to manipulate and manage the PDU in any manner necessary. In some instances, however, the internet PDU (data and headers) remains unchanged as it is transmitted through the subnet. In FIG. 1, it emerges at the gateway where it is processed through the lower layers 114 and passed to the IP (network) layer 112. Here, routing decisions are made based on the destination address provided by the host computer.
After these routing decisions have been made, the PDU is passed to the communications link connected to the appropriate subnetwork (comprising the lower layers). The PDU is re-encapsulated into the data link layer frame and passed to the next subnetwork 116, where it finally arrives at the destination host.
The destination (host B) receives the traffic through its lower layers and reverses the process that transpired at host A; it de-encapsulates the headers by stripping them off in the appropriate layer. The header is used by the layer to determine the actions it is to perform; the header therefore governs the layer's operations.
The PDU created by the file transfer application in the application service layer is passed to the file transfer application residing at host B. If host A and B are large mainframe computers, this application is likely an exact duplicate of the software at the transmitting host. The application might, however, perform a variety of functions, depending on the header it receives. It is conceivable that the data could be passed to another end-user application at host B, but in many instances the user at host A merely wants to obtain the services of a server protocol, such as a file transfer or email. If this is the case, it is not necessary for an end-user application process to be invoked at host B.
To return the retrieved data from the server at host B to the client at host A, the process is reversed. The data is transferred down through the layers in the host B machine, through the network, through the gateway, to the next network, and up the layers of host A to the end-user.
In Frame Relay networks, the operation of and the various services provided by the data link layer in the diagram of FIG. 1 are governed in accordance with the implementation agreements promulgated by the Frame Relay Forum (FRF). FRF. 16.1 (incorporated herein by reference above) describes techniques which enable multiple serial links to be aggregated into a single bundle of bandwidth, i.e., Multi-Link Frame Relay (MFR). More specifically, MFR enables the creation of a virtual interface called a bundle or bundle interface. The bundle interface emulates a physical interface for the transport of frames. The Frame Relay data link runs on the bundle interface, and Frame Relay virtual circuits are built upon it.
A bundle is made up of multiple serial links, called bundle links. Each bundle link within a bundle corresponds to a physical interface. Such a physical interface might correspond, for example, to a T1 or T3 interface. Bundle links are invisible to the Frame Relay data-link layer. As a result, Frame Relay functionality cannot be configured on these interfaces individually. Rather, the desired Frame Relay functionality must be configured on the bundle interface. Bundle links are visible to peer devices. The local router and peer devices exchange link integrity protocol control messages to determine which bundle links are operational and to synchronize which bundle links should be associated with which bundles.
For link management, each end of a bundle link follows the MFR Link Integrity Protocol and exchanges link control messages with its peer (the other end of the bundle link). To bring up a bundle link, both ends of the link must complete an exchange of ADD_LINK and ADD_LINK_ACK messages. To maintain the link, both ends periodically exchange HELLO and HELLO_ACK messages. This exchange of hello messages and acknowledgments serve as a “keep-alive” mechanism for the link. If a router is sending hello messages but not receiving acknowledgments, it will resend the hello message up to a configured maximum number of times. If the router exhausts the maximum number of retries, the link status is considered “DOWN” (unoperational).
Each link's status is considered “UP” (operational) when the corresponding link on the peer device acknowledges that it will use the same link for the bundle. The link's status remains “UP” when the corresponding link acknowledges the hello messages. The bundle interface's status becomes “UP” when at least one of its link has its status “UP.” The bundle interface's status goes “DOWN” when the last bundle link is no longer in the “UP” state. This behavior complies with the class A bandwidth requirement defined in FRF.16.1.
Multi-Link Frame Relay provides load balancing across the bundle links within a bundle. If a bundle link chosen for transmission happens to be busy transmitting a long packet, the load balancing mechanism can try another link, thus solving the problems seen when delay-sensitive packets have to wait.
In addition, by combining multiple physical interfaces into a bundle, a Frame Relay interface may be designed with more bandwidth than is available from any single physical interface. For example, many new network applications require more bandwidth than is available on a T1 line. One option is to invest in a T3 line. However, T3 lines can be expensive and are not available in some locations. MFR provides a cost-effective solution to this problem by allowing multiple T1 lines to be aggregated into a single bundle of bandwidth.
Finally, MFR can provide greater service resilience when multiple physical interfaces are provisioned as a single bundle. That is, when a link fails the bundle continues to support the Frame Relay service by transmitting across the remaining bundle links.
One of the challenges associated with the MFR protocol is that it does not allow the physical relocation of bundle links from one bundle interface to another without manual intervention by the personnel of the service provider managing the intervening network, e.g., AT&T. An example referring to the block diagram of FIG. 2 will be illustrative of the disadvantages associated with such an approach.
In this example, bundle interface A of device 202 (including bundle links A1, A2, and A3) is logically connected with bundle interface B of device 204 (including bundle links B1, B2, and B3). When bundle links A1, A2, and A3 are physically connected to bundle links B1, B2, and B3, respectively, the MFR protocol begins exchanging control messages between the peer end points which contain the respective bundle and link identifications (BIDs and LIDs). For example, control messages sent from A1 to B1 contain the BID corresponding to A and the LID corresponding to A1. Similarly, control messages sent from B1 to A1 contain the BID corresponding to B and the LID corresponding to B1. Through the mutual exchange of these control messages, bundle interfaces A and B learn their peers, store the bundle and link identification information, and become operational.
If bundle links A1, A2, and A3 of bundle interface A are subsequently physically relocated to bundle links C1, C2, and C3 of bundle interface C (of device 206), the subsequent control messages between these links will indicate to the links of bundle interface A that the peer has changed from bundle interface B to bundle interface C. And because of the mismatch between the bundle IDs received on bundle links C1, C2 and C3 and the previously stored values, all of the links (and therefore the bundle interfaces) remain in the “DOWN” state. Thus, bundle interfaces A and C do not become operational.
Currently, the only way to effect such a physical relocation of links is by notifying the service provider managing the intervening network so that the necessary reconfiguration may be performed manually by the service provider's personnel. This involves the re-provisioning of the links. This typically requires the representative of the service provided to log onto a command line interface (CLI) and manually delete the bundle and link identification information associated with the previously learned peer interface(s), and then add the new links to begin synchronization with the new bundle. The disadvantages associated with this approach are manifest.